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Abstract 

The canonical ensemble describes an open system in equilibrium with a heat bath of fixed 
temperature. The probability distribution of such a system, the Boltzmann distribution, is derived 
from the uniform probability distribution of the closed universe consisting of the open system 
and the heat bath, by taking the limit where the heat bath is much larger than the system of 
interest. Alternatively, the Boltzmann distribution can be derived from the Maximum Entropy 
Principle, where the Gibbs-Shannon entropy is maximized under the constraint that the mean 
energy of the open system is fixed. To make the connection between these two apparently distinct 
methods for deriving the Boltzmann distribution, it is first shown that the uniform distribution 
for a microcanonical distribution is obtained from the Maximum Entropy Principle applied to a 
closed system. Then I show that the target function in the Maximum Entropy Principle for the 
open system, is obtained by partial maximization of Gibbs-Shannon entropy of the closed universe 
over the microstate probability distributions of the heat bath. Thus, microcanonical origin of 
the Entropy Maximization procedure for an open system, is established in a rigorous manner, 
showing the equivalence between apparently two distinct approaches for deriving the Boltzmann 
distribution. By extending the mathematical formalism to dynamical paths, the result may also 
provide an alternative justification for the principle of path entropy maximization as well. 
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I. INTRODUCTION 



Although canonical ensemble is widely used for computing various thermodynamic quan- 
tities, the most fundamental definitions and postulates in statistical mechanics are all for- 
mulated for a microncanonical ensemble, which describes a system in isolation. In contrast, 
the canonical ensemble describes a system in thermal contact with a heat bath with a fixed 
temperature. This ensemble can be considered as a special limit of the microcanonical en- 
semble, where the system consists of weakly coupled two subsystems and one of them is 
much larger than the other. Therefore, one might say that the microcanonical formulation 
of statistical physics is more fundamental, and the canonical formalism can be derived from 
the former. 

The basic postulate of the statistical mechanics for an isolated system, with fixed param- 
eters specifying the macroscopic state, is that all the microscopic states consistent with these 
macroscopic parameters have equal probability of being occupied by the systemjl sj]. The 
Boltzmann distribution, the probability distribution of microstates of canonical ensemble, 
is then obtained by dividing the system into the system of interest and the heat bath, the 
latter being much larger than the former, and then summing over all the microstates of the 



latter [ 

An alternative derivation of the Boltzmann distribution is from the so-called Maximum 
Entropy principlejl, 2, 5, 6]. Here the Gibbs-Shannon entropy H = —J2iPi hi Pi is maximized 
under the constraint that the expectation value of energy, J2iPiEi, is fixed to a certain value, 
where Pi and E$ are the occupation probability and the energy of the microstate labeled as i. 
No transparent connection between the two approaches has been established, especially why 
fixing the temperature of the heat bath is equivalent to fixing the mean energy of the open 
system. In fact, the temperature of the heat bath determines the most probable value of 
open system energy, which is determined by the condition that the derivative of the system 
entropy with respect to energy should be equal to the inverse of the heat bath temperature. 
In the limit of infinite system size, the relative fluctuation of energy from the expectation 
value becomes tiny, and the expectation value of energy becomes essentially the same as 
the most probable value of energy, but the equivalence breaks down once we consider an 
open system with a small size. In this era where small systems with nano-scale sizes can be 
probed with ease and mesoscopic systems have become subject of interest [3], it is important 



to formulate statistical physics without taking the limit of infinite system size. In this work, 
only the size of the heat bath is taken to infinity by definition, but the system size will be 
kept finite. 



Providing the microcanonical origin for the Maximum Entropy principle is important 
because the latter is applied also to dynamical systems jg- 14]. In this case, the probability 
appearing in the Gibbs-Shannon entropy is the probability of each dynamical path, and the 
entropy is maximized under the constraint that the expectation value of quantities such as 
flux is held fixed, to obtain Boltzmann-like distribution for the path probability. Since the 
mathematics of the present work can be reinterpreted in the context of dynamical systems, an 
alternative justification for such a path probability distribution can be provided, which does 
not have as a firm foundation as the Boltzmann distribution in the case of the equilibrium 
system. 



In this work, I show that the Maximum Entropy principle for the open system simply 
follows from the Maximum Entropy principle applied to the closed universe consisting of 
the open system and the heat bath, by performing partial maximization of the Gibbs- 
Shannon entropy of the closed universe, over the probability distributions of microstates 
of the heat bath. The organization of the paper is as follows. I first explain the entropy 
maximization for the closed system, and how uniform distribution is obtained as the result. 
The standard derivation of Boltzmann distribution from the uniform distribution by dividing 
the closed universe into the open system and the heat bath, is also briefly reviewed. It is then 
shown that the target function to be used for entropy maximization procedure of an open 
system, the Gibbs-Shannon entropy plus the Lagrange term constraining the mean energy 
of the open system, is derived by performing a partial maximization of the Gibbs-Shannon 
entropy of the closed universe, over the probability distribution of microstates of the heat 
bath. Therefore, the microcanonical origin of the entropy maximization procedure for an 
open system is established, showing that equilibrating an open system with a heat bath 
of fixed temperature is equivalent to constraining its mean energy. Thus, two apparently 
distinct ways of deriving the Boltzmann distribution are shown to be the same thing. 
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II. MAXIMUM ENTROPY PRINCIPLE FOR CLOSE SYSTEM AND THE UNI- 
FORM DISTRIBUTION 



The basic postulate of the statistical mechanics is that all the microstates, consistent 
with the given macroscopic parameters, have the same probability of being occupied by the 
system. For concreteness, let us suppose that the total energy E of the system is given. 
Then the uniform distribution for the microcanonical ensemble describing this system is 
expressed as: 

See 

where Q(E) is the total number of microstates with energy E. It is to be understood that all 
the other parameters such as volume or particle number are also fixed, and states with any 
other values of these parameters are excluded when counting the state. Only the constraint 
of fixed energy, which is of interest here, will be explicitly treated with the Kronecker 
delta functions, and any other macroscopic parameters will be suppressed throughout the 
manuscript for notational simplicity. 

Rather than being treated as a starting assumption, the uniform distribution in Eq.([T]) 
can be considered as a consequence of maximizing the Gibbs-Shannon entropy 

H({ Pl }) = -J2^Pi (2) 

i 

under the constraint that the energy value Ei of any microstate i with nonzero probability 
should be E: 

5>(l-<^) = (3) 

i 

along with the normalization condition 

5> = 1. (4) 



For notational simplicity, we assume discrete energy levels in this work. When the energy level is contin- 
uous, it is more natural to count the number of states with energy values lying between E and E + dE 
with some small number dE, and 8E,Ei in Eq.(TTJ) should be replaced by Q(Ei — E)6(E + dE — Ei), with 
corresponding minor modifications of the mathematics in the following sections. 
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These constraints are imposed using Lagrange multipliers, so that the function 



H({pi}) + AJ>(1 - 5 EiE ) + vC£p* ~ 1) 

i i 

= Pi ln Pi + A$^Pi(l - 5 EiE ) + v(^Pi ~ 1) 

i i i 

is to be varied with respect to {p%}, A, v and then set to zero: 

- log Pi - 1 + A(l - 5 EiE ) + v = 

i 

I>-i = o 



(5) 



(6) 



The first line of Eq. (jH]) yields 



p l = e y ~ 1 (Ei — E) 



Pi 



{Ei + E), 



(7) 



and the second and the third line determine the values of the Lagrange multipliers, yielding^ 



v = l-lnQ(E) 

X = — oo, 



(8) 



resulting solution being the uniform distribution Eq.([T]). The target function after the 
maximization is the Boltzmann entropy^ 



S(E) = \ntt(E). 



(9) 



under macroscopic constraints was 
and proposed as the prescription for 



The maximization of the Gibbs-Shannon entropy 
termed as Maximum Entropy principle by Jaynesjl 
obtaining the equilibrium distribution, but the principle was applied mainly to open system, 
most probably because the uniform distribution that results for a close system is considered 
as trivial. However, since the closed system can be considered as more fundamental and any 



To avoid an infinite Lagrange multiplier, one may use r defined as r 



instead of A. 



3 In this work, the Boltzmann constant, which is just a unit conversion factor introduced for a historical 
reason, will be set to one, assuming that the temperature is measured in the same unit as the energy and 
the entropy is a dimensionless quantity. 



result on open system can be derived as a special limit of a subsystem of a closed universe, 
it is natural to expect that the Maximum Entropy principle for an open system should be 
derivable from that for the closed one. 



Once the uniform distribution for a closed system is obtained, the Boltzmann distribution 



3ath immediately follows, as can be found 
|, |4|. The open system and the heat bath 



for a open system in thermal contact with a heat 
in any standard textbook on statistical physics jl|, 
form a closed universe, the probability distribution of microstates being the uniform one. 
The probability distribution of microstates for the open system only, which is a subsystem 
of the closed universe, is obtained as a marginal probability, where it is easy to see that the 
probability for a given microstate i is proportional to the number of heat bath microstates 
consistent with the system energy Ef. 

Pi oc fi ba th(£tot - Et) = exp [S hath (E tot - Ei)] ~ exp [S hath (E tot ) - [3Ei] (10) 

where E tot is the total energy of the closed universe, which is a fixed constant, and the first 
order approximation in the last term comes from the fact that heat bath is much larger than 
the open system so that the value of temperature T = ^ bath is a fixed constant that 
does not depend on the open system energy E. The standard assumption of weak coupling, 
that the energy is additive and the number states can be factorized for a given value E{, is 
being used. Since Sb a th(-£aot) is a constant, Eq. fTDjl is the Boltzmann distribution. 

The same Boltzmann distribution can also be obtained by applying the Maximum En- 
tropy principle directly to the open system, under the constraint that the expectation value 
of the fluctuating energy, J^PiEi, is fixed to a particular value e. Introducing the Lagrange 
multiplier, the function to extremize is now 

H({Pi}) ~ 0(Y,Pi E i - e ) + "(£ft - !) 

i i 

= -j^Pi^Pi - /?(£p^ - e ) + v(Epi - !)■ ( n ) 

i i i 

The resulting solution takes the form of the Boltzmann distribution: 



where (3 is determined by the condition 



(13) 



However, the connection of the microcanonical derivation and the entropy maximization 
approach is not clear at all, since j3 here is introduced as the Lagrange multiplier for con- 
straining the mean energy, whereas (3 appearing in the microcanonical derivation is the 
inverse of heat bath temperature, the derivative of the heat bath entropy with respect to 
energy. 

In the next section, it is shown that the Maximum Entropy principle for the open system 
is derived in the process of a two-step maximization of Gibbs-Shannon entropy for the entire 
closed universe, and the target function including the Lagrange term for constraining the 
energy expectation value of the open system is obtained by partial maximization of the 
Gibbs-Shannon entropy of the closed universe, over the microstate probability distribution 
of the closed universe. 



III. MAXIMUM ENTROPY PRINCIPLE FOR OPEN SYSTEM, DERIVED BY 
PARTIAL MAXIMIZATION OF THE ENTROPY OF THE CLOSED UNIVERSE 

In order to derive the target function of entropy maximization procedure for open system, 
Eq.f llip . we first write down the Gibbs-Shannon entropy for the closed universe consisting 
of an open system and a heat bath, along with the Lagrange multiplier terms for the energy 
and normalization constraints: 

H to t({Pia}) + v(^lPia - 1) + A^p io (l - S Ei+EajE ) 
i,a i,a 

= -^Pia^Pi a + ls(Y;Pia ~ 1) + A^p ia (l ~ 5 El+Ea , E ) (14) 
i,a i,a i,a 

where indices i and a labels the microstates of the open system and the heat bath, respec- 
tively. If the number of possible microstates in the open system and the heat bath are 
denoted as A and B, we see there are AB components of the variables Pi a , but the normal- 
ization condition J2iaPia = 1 reduces the number of independent components to AB — 1. We 
now change variables from p ia to pi = J2aPia and p(a\i) = Pi a /Pi- The number of components 
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of pi and p(a\i) are A and AB, but again the normalization conditions 



5> = 1 

EKa|*) = l (i = l-..A) (15) 

a 

reduce their number of independent components to A — 1 and AS — A respectively, mak- 
ing the total number of independent components to AB — 1. Thus, to ensure the correct 
number of independent variables and maintain full equivalence, the normalization condi- 
tions (fT5|) should be implemented using the Lagrange multipliers. The target function for 
maximization, expressed in terms of the new variables, is: 

- EK a K)fi ln (p(a|*)Pi) + E ^(E^( a l ! ) - !) + ME?* - !) + ^EPiPfaWC 1 ~ Sjs i+ E *> E ) 

i,a i & i i,a 

= - ^Pilnpi - ^p(a\i)pi\\ip(a\i) + E ^(EpW*) ~ l ) 

i i,a i a- 

~ l ) + ^EPiPHi)( 1 - <W£a,s) (16) 

where the conditions ( fl5l) was used in producing the second line. Since the open system is 
the object of interest, we perform the maximization in two steps. First, the variation with 
respect to p(a\i) , z/j, and A is performed, to eliminate them for given values of pj. Then the 
target function is maximized with respect to the remaining variables Pi and \i at the second 
step. 

Taking variation of the target function in Eq. flTB]) with respect to p(a\i) , fj, and A and 
setting them to zero, we get: 

- pi \np(a\i) -Pi + Vi + \pi{l - 5 Ei+Ea>E ) = (17) 

Ef(a|<) = l ( 18 ) 

a 

J2PiP(a\i)(l-5 Ei+Ea>E ) = 0. (19) 

i,a 

From Eq. ffTTl) we get 

p(a\i) = exp(— - 1) (E a = E tot - E*) 
Pi 

p(a\i) = exp(— - 1 + A) {E a ^ E tot - E { ) . (20) 

Pi 
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The constraints Eq. (|T9|) and Eq.( TT8|) fixes the values of A and z/j so that we finally obtain 



p(o|!) = m^E~) (21) 

Substituting Eq. (l2T|) into Eq. (|T6|) . we now get 

-5>in ft + E^Jt^rty ln Wot - Ek) + M5> - 1) 

? i,a \ tot 2/ 

^-EPilnft + EPi^^tot-^+ME^- 1 )- ( 22 ) 

i j i 

Now we use the fact that heat bath is much larger than the open system so that lnQ(E tot — 
Ei) ~ \ntt(E tot ) - f to get 

H{{Pi}) +fi(EPi ~ !) = -E^ m ^ - /^E^ + ME^ - !)■ (23) 



where (3 = 1/T and an irrelevant constant term was dropped. Therefore, we see that the 
target function for the Maximum Entropy procedure for an open system, Eq.f llip is now 
reproduced. 



This also shows that providing the open system with a thermal contact with a heat bath of 
fixed temperature, is equivalent to constraining the expectation value of energy. To see this 
point more intuitively, note that the actual energy levels of the heat bath are not relevant, 
as long as the total energy of E tot is maintained at the value satisfying 

[dS hath (E)/dE} E=Etot = 1/T. (24) 

Then it is convenient to replace the heat bath with N replica of the open system, where N 
is to be taken to infinity. Maintaining the condition Eq.( T2"4"]) then corresponds to fixing the 
average energy of iV + 1 open systems, E = E tot /N. Since there are N + 1 identical open 
systems, as iV — > oo, E becomes equal to the expectation value of energy for each open 
system (E) = ^ ViE% due to the law of large numbers. 

9 



IV. FREE ENERGY MINIMIZATION PRINCIPLE 



We note that after multiplying H({pi}; T) by T and inverting the sign, we obtain 

HM;T) = Y.P l E l -TH{{p}) = Y,PiEi + Tj2Pi^Pi, (25) 

i 

which is to be minimized with respect to Pi under the normalization constraint J2iPi 



I . In fact, this function also appears in the literature [15j, and since J^PiEi is the energy 
expectation value and H({p}) is an entropy, F({pi};T) is considered as a Helmholtz free 
energy, and the corresponding minimization principle as the free energy minimization, but 
its microcanonical origin has never been explained so far. 

This is to be contrasted with the minimization of thermodynamic Helmholtz free energy, 



F(E;T) = E -TS(E) (26) 

which is a function of the open system energy E. The minimization of Eq. (1261) with respect 
to E yields the most probable value of E of the open system at equilibrium. On the 
other hand, the minimization of F({pi};T) yields the equilibrium probability distribution 
of microstatesfl 



V. DISCUSSION 



There are two apparently distinct methods for deriving the Boltzmann distribution, one 
by considering the microcanonical ensemble of the closed universe consisting of the open 
system and the heat bath, and the other by maximizing the Gibbs-Shannon entropy of the 
open system under the constraint of fixed value of mean energy. In this work, it was shown 
that the Maximum Entropy procedure for the open system arises in the process of two- 
step maximization of the Gibbs-Shannon entropy of the closed universe, thus showing the 
equivalence of providing contact with the heat bath and constraining the energy expectation 
value. From this viewpoint, the microcanonical derivation and Maximum Entropy derivation 
of the Boltzmann distribution are the same thing. In the former, the Gibbs-Shannon entropy 

4 Note that F(E, T) and F({pi}; T) are distinct even after the minimizations. For the former, the minimum 
occurs at E m i n (T) satisfying dS / dE\ EuLin {T) = 1/T and the resulting function A(T) = F(E min (T),T) is 
the Legendre transform of S(E). On the other hand, the minimization of F({pi};T) yields A(T) = 
—Tin V\ eT^ Ei = — T\n^2 E cxp(—(3F(E; T)). They are equal only in the limit of infinite system size. 
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of the closed universe is maximized in one step, resulting in the uniform distribution of the 
microstate, which is then divided into open system and the heat bath to obtain the marginal 
probability distribution for the open system. In the latter, the maximization is performed 
in two steps, where partial maximization is performed over the microstates of the heat bath 
first, to obtain the target function expressed in terms of the probability distribution of the 
open system. Since it is a well-known fact that one-step or multi-step maximization of a 
function should yield the same result, it is no surprise that the same Boltzmann distribution 
is derived. 



Although energy was the main focus in this work for the sake of concreteness, the ar- 
gument can be generalized to any other macroscopic parameters such as volume or particle 
number. The entropy maximization procedure where the expectation value of a macroscopic 
parameter a is constrained, results in a Boltzmann form of distribution p, oc e~ 7C ^ with the 
corresponding Lagrange multiplier 7, and the microcanonical origin of this procedure is the 
entropy maximization for closed universe, consisting of the open system and the bath with 
fixed value of 7 = dS/da. 



The case of particular interest is a dynamical system, where the probability distribution 
of dynamical paths is obtained by maximizing the Gibbs-Shannon entropy of path entropy, 
under the constraint on the expected value of quantities such as flux jg- 14]. The argument of 
the present work provides an alternative justification, but it should be noted however, that 
the total "universe" consisting of the open system and the "flux bath" cannot be interpreted 
as a closed universe, since a system with a nonzero flux is always a system driven by an 
external influence. In the case of dynamic system, one just considers a large system with 
a fixed value of total flux. Then, in the absence of any other additional information, the 
probability of each micropaths consistent with the constraint is equally probable, due to 
the Maximum Entropy principle. Then if we restrict our interest to the subsystem of this 
large system, the marginal probability distribution of the paths of the subsystem would be 
the one obtained by maximizing the path entropy under the constraint of fixed expectation 
value of the flux. 
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